Evaluation of online text‐based information resources of gynaecological cancer symptoms

Abstract Background Gynaecological cancer symptoms are often vague and non‐specific. Quality health information is central to timely cancer diagnosis and treatment. The aim of this study was to identify and evaluate the quality of online text‐based patient information resources regarding gynaecological cancer symptoms. Methods A targeted website search and Google search were conducted to identify health information resources published by the Australian government and non‐government health organisations. Resources were classified by topic (gynaecological health, gynaecological cancers, cancer, general health); assessed for reading level (Simple Measure of Gobbledygook, SMOG) and difficulty (Flesch Reading Ease, FRE); understandability and actionability (Patient Education Materials Assessment Tool, PEMAT, 0–100), whereby higher scores indicate better understandability/actionability. Seven criteria were used to assess cultural inclusivity specific for Aboriginal and Torres Strait Islander people; resources which met 3–5 items were deemed to be moderately inclusive and 6+ items as inclusive. Results A total of 109 resources were identified and 76% provided information on symptoms in the context of gynaecological cancers. The average readability was equivalent to a grade 10 reading level on the SMOG and classified as ‘difficult to read’ on the FRE. The mean PEMAT scores were 95% (range 58–100) for understandability and 13% (range 0–80) for actionability. Five resources were evaluated as being moderately culturally inclusive. No resource met all the benchmarks. Conclusions This study highlights the inadequate quality of online resources available on pre‐diagnosis gynaecological cancer symptom information. Resources should be revised in line with the recommended standards for readability, understandability and actionability and to meet the needs of a culturally diverse population.


| INTRODUCTION
Gynaecological cancers start in the female reproductive system or genitals and include cervical, ovarian, fallopian tube, primary peritoneal, uterine/endometrial, vaginal and vulvar cancer.An estimated 6700 women will be diagnosed with gynaecological cancer in Australia in 2022 with a projected age-standardised incidence of 42 cases per 100,000. 1 Uterine, ovarian and cervical cancers have remained in the top 20 most commonly diagnosed cancers among Australian females for 40 years (1982 and 2022). 2 Between 2014 and 2018, 71% of Australians diagnosed with gynaecological cancer were alive 5 years after diagnosis, with uterine cancer having the highest 5-year survival rate (83%) and ovarian cancer having the lowest (48%).Five-year cancer survival rates decline with more advanced disease at diagnosis.The burden of gynaecological cancers is disproportionately shared among the Australian population.National data indicate that Aboriginal and Torres Strait Islander women are more likely to be diagnosed and die from gynaecological cancers than non-Indigenous women. 1,3,4hile Australia has a national screening program for cervical cancer, timely diagnosis of other gynaecological cancers largely relies on the early identification of symptoms.The Model of Pathways to Treatment describes the events and processes that occur along the pathway to cancer diagnosis, identifying potential barriers and facilitators to diagnosis and treatment across four intervals: appraisal, help-seeking, diagnostic and pre-treatment. 5Here we are interested in the intervals prior to diagnosis.Documented barriers for appraising gynaecological cancer symptoms include experiencing non-specific symptoms (e.g.gastrointestinal symptoms), normalisation of symptoms (e.g.attributing them to age, a side effect of hormonal contraception), self-management of symptoms (e.g.lifestyle changes), a perception of good health (e.g.7][8] Help-seeking may be delayed for several reasons, such as symptom embarrassment, concerns about potential judgement from providers (including wasting the doctor's time), access to care (including financially), competing time demands (e.g.work, carer duties) and unmet information needs (i.e.][8][9] Although less is known about the facilitators for appraisal and help-seeking, a potentially modifiable factor is awareness that symptoms may be serious; knowledge which may be sought via available health information. 7,8n fact, a key goal of the Australian Cancer Plan is to improve and build the cancer health literacy of the Australian population. 10Health literacy involves the ability of an individual to navigate the healthcare system and make health decisions; this includes the use of electronic technology. 10Over 80% of Australians go online first for health information and 73% have already used the internet to research a health issue before visiting their doctor, although only 6% manage to find an online health source that they trust. 11,12The Australian population is diverse and, therefore, health information resources need to be readable, understandable and actionable by people from across different cultural and socio-economic backgrounds.Ideally, such resources would be co-designed with end users to ensure resources are more likely to meet their needs.End users may include a range of stakeholders including healthcare consumers, families and health professionals. 13fter a review of the current literature, there has been no formal assessment of the availability and accessibility of information resources on gynaecological cancer symptoms for the Australian population, nor specifically the availability of resources that are culturally relevant to Aboriginal and Torres Strait Islander people, who carry the greatest burden of gynaecological cancers.Therefore, this study aims to identify and evaluate the quality of Australian online text-based patient information resources regarding gynaecological cancer symptoms.

| Positionality of authors
The research team acknowledge that our individual and shared values, perspectives and experiences can shape research processes.Therefore, we recognise the value of reflexivity in research.The authorship team includes three non-Indigenous Australian researchers, including two midcareer and one early career public health researchers; two of whom are epidemiologists.Content areas of expertise include gynaecological cancer care, supportive care in cancer, information needs in people affected by cancer, Indigenous health, health services gaps and health inequities.The project team also included an Aboriginal research assistant.

| Search strategy and resource selection
Two search strategies were employed to identify relevant health information resources about gynaecological cancer symptoms: (1) a targeted website search and (2) a Google search.
Targeted website search: A comprehensive list of Australian government and non-government health websites that provide information, advice or services related to cancer, gynaecological health or women's health was compiled by the research team (Table 1).Multiple searches were conducted within each website using their inbuilt search function and imputing one keyword per search to identify relevant sub-pages or embedded documents (collectively referred to as resources).Keywords included gynaecology-, cancer-and symptom-specific terms (Table 2).Here less than 50 resources were identified from a single search, all were assessed for eligibility.Otherwise, eligibility continued to be assessed beyond the first 50 resources until either all had been assessed or 20 consecutive resources were deemed ineligible.This pragmatic approach was adapted to ensure the most relevant resources were considered, while also limiting time spent on irrelevant resources. 14These searches were conducted between August and September 2022.
Google search: An advanced Google search was conducted on 21 November 2022.This search engine was selected as it is the most used in Australia. 14Results were limited to the English language and the region of Australia.The results displayed a title, web link and description, referred to as a 'snippet.'In this study, 10 snippets were listed on each results page.The first five pages of snippets were screened to identify potentially relevant resources, and screening continued until there were two consecutive pages that did not contain any potentially relevant resources.Snippets were screened to identify potentially relevant resources, which were then retrieved and reviewed against the eligibility criteria.
Eligibility criteria were systematically applied.Online text-based resources were deemed eligible if they were published by a reliable Australian health authority, such as a government or non-government health organisation or health service, were consumer-focused and included any level of information about gynaecological cancer symptoms.Other information resources, such as media articles, research papers and opinion pieces (columns, blogs) were excluded, as were healthcare provider-focused information resources.Screening was completed by one investigator (CS) and, where there was uncertainty, discussed with another investigator (TD and/or AD) to determine eligibility.

| Evaluation of patient information resources
A data extraction form was created and piloted by two investigators (TD and CS) to ensure that key information to

| Readability
Two measures of readability were used in this evaluation: (1) the Simple Measure of Gobbledygook (SMOG) estimates the years of education (i.e.school grade) required to comprehend the words used in the text based on the number of words with three syllables, 2,15,16 and the Flesch Reading Ease (FRE) which calculates a score based on sentence length and number of syllables to determine the reading difficulty (score 0 = most difficult, score 100 = easiest). 15,17,18Multiple readability tools are often used to limit bias, as there is variability in the leniency of readability tools. 19A grade 8 reading level is recommended for health resources in Australia 20 and an FRE score of 60 or higher (60-69 standard/average to 90-100 very easy to read) is deemed readable to the general population in Australia. 14,21As such, these benchmarks were used in this evaluation to categorise the resources as readable or not.
An online calculator was used to conduct the readability evaluations (https:// reada bilit yform ulas.com).The online calculator requires a minimum of 150 words and a maximum of 3000 words from the text. 22Where possible, all text relevant to gynaecological symptoms was included.Where relevant text exceeded 3000 words, extracts from at least three different sections were included, towards the start, middle and end of the relevant text.Where the relevant word count was less than 150 words, the relevant text was copied until the minimum word count was reached.

| Understandability and actionability
The Patient Education Materials Assessment Tool (PEMAT) was used to systematically evaluate understandability and actionability of the included resources. 23nderstandability refers to the ability of users of diverse backgrounds and health literacy levels to process and explain key messages of the resources.Actionability refers to the ability of users of diverse backgrounds and health literacy levels to identify action steps or advice for the benefit of their own health based on the information provided in the resource.
4][25] As per the PEMAT Handbook, 23 one investigator answered each item as yes (score = 1) or no (score = 0) and the item scores were summed and divided by the total number of items to produce a score for the understandability and actionability constructs separately.These scores were expressed as a percentage (ranging 0% = not at all understandable/actionable and 100% = easily understood/ actionable).A score of 70% and over identified a resource as 'actionable' and 'understandable'. 26

| Cultural inclusivity
There is currently no validated measure of the cultural relevance of patient health information resources to Aboriginal and Torres Strait Islander people.To our knowledge, only one approach has been used previously. 27A team of Aboriginal and Torres Strait Islander and non-Indigenous researchers (including investigator AD) with experience working in Indigenous health research devised seven items to assess patient information resources for cultural inclusivity.While no benchmark was provided to aid in a formal evaluation, the measure provides an indication of whether resources could be considered specifically for or inclusive of Aboriginal and Torres Strait Islander people.The seven items were As this tool has not been validated and no benchmark determined, we somewhat arbitrarily categorised resources into three groups: not inclusive (less than 3 items were met), moderately inclusive (3-5 items were met) and inclusive (6-7 items were met).

| RESULTS
As illustrated in Figure 1, 23,943 potential resources were initially identified through the two search strategies.The snippets of each were assessed against the eligibility criteria and 23,092 were initially excluded.A full review of the remaining 851 resources was conducted, which resulted in a further 611 resources being excluded as duplicates and 131 excluded for not meeting eligibility criteria.The remaining 109 resources are included in this evaluation.Resources were from 25 organisations, agencies or services, including government (n = 8), nongovernment (n = 8) and public or private health services (n = 9) (Table 3).Fourteen of the 109 resources were available in PDF printable format, and the remaining 95 were print-friendly web pages.All were presented in colour.Weblinks to included resources are available in Data S1.

| Content summary
A summary of the evaluation of included resources is available in Table 3. Three-quarters (76%) of resources focused on gynaecological cancers, 9% on cancer symptoms, 9% on gynaecological health and 6% on general health.The majority (78%) of resources provided information on more than one symptom group.The most common symptom group identified was abdominal discomfort (75%), followed by abnormal/persistent bleeding (72%), general symptoms (59%), changes in bowel and/or bladder habits (58%) and vulval symptoms (48%).

| Readability
The mean SMOG score was 10.4 (range 5.6-17.1),equating to a required reading level of grade 10, and the mean FRE score was 47.9 (range 11.0-77.8)denoting resources were on average 'difficult to read' (Tables 3 and 4).The a priori benchmarks were set at a grade 8 reading level based on the SMOG and a FRE score of 60 or higher.In this study, 16% of resources received a SMOG score equivalent to grade 8 or below, 18% of resources had a FRE score of 60 or higher (standard/average or easier) (Table 3) and 12% of resources were deemed readable using both SMOG and FRE measures (Table 4).

| Actionability
The mean PEMAT actionability score was 48.1% (range 0%-80%) (Table 3).Only 13% of resources received an actionability score of 70% or higher, thus meeting the benchmark set for this evaluation (Table 4).The most common actionable advice provided to the readers was to consult their general practitioner (GP) if symptoms arose and/or persisted.

| Cultural inclusivity
Five resources were considered moderately culturally inclusive for Aboriginal and Torres Strait Islander people (Table 4).Four of these resources were explicitly Aboriginal and Torres Strait Islander-focused and related to cervical cancer, uterine cancer, cancer (Cancer Councils of Australia, resources #50, #55, #72) and ovarian cancer [Cherbourg Regional Aboriginal and Islander Community Controlled

| DISCUSSION
This study identified and evaluated the quality of Australian, online text-based, information resources on gynaecological cancer symptoms.Given that 80% of Australians access health information online, 11 it is important for women to be able to source online information that is readable, understandable, actionable and culturally inclusive.While there was no shortage in the quantity of resources (n = 109), the quality varied considerably with just six resources meeting three (i.e.readable, understandable, actionable) of the four (plus cultural inclusivity) predetermined benchmarks.
Although a significant number of online information resources for gynaecological cancer symptoms are published by Australian health authorities, not all symptom groups were addressed equally.Unexplained vulval symptoms were the least described (<50% of resources mentioned vulval symptoms such as a lump, ulceration, bleeding, skin colour changes, persistent itching, burning or soreness).Women diagnosed with gynaecological cancer, and vulval cancer in particular, have identified information barriers including not knowing where to go for information, not knowing what to ask, and a preference for information from sources other than the internet, highlighting the importance of ensuring quality health resources in different formats to educate and support the diagnostic process. 7,8,28verall, the resources identified were rated as 'difficult' to read by the general public with the majority not meeting the Australian recommended guidelines for readability of health resources.0][31] In fact, Australian heath websites have been found to be written at two to four grades higher than recommended; indeed, information on bowel and breast cancer is significantly harder to read than other conditions such as anxiety, diabetes and obesity. 32here was a high standard of understandability among the evaluated resources with the majority meeting this benchmark. 26However, there is room for further improvement as few resources use visual aids.Integrating visual aids supports content to be more easily understood, clearly directs readers to important information and increases the aesthetic appearance of information. 23Furthermore, the use of visual aids in health information can specifically benefit patients with limited health literacy. 23Conversely, there was a lack of actionable information provided with only 13% of resources meeting the benchmark. 26The most common action recommended was for readers to consult their GP if they are experiencing any symptoms.
The findings illustrate limited culturally inclusive resources, with only four resources written for Aboriginal and Torres Strait Islander populations.Australian research has highlighted the importance of individualising cancer information by taking into consideration demographic factors and attitudes 33 ; although this finding is specific to rural cancer survivors, the conclusions are also relevant to enhancing the inclusion of Aboriginal and Torres Strait Islander people in healthcare.Aboriginal and Torres Strait Islander women experience higher incidence and mortality rates for gynaecological cancers than non-Indigenous women, particularly for cervical cancer, although no national data are available on vaginal and vulvar cancers. 34ulturally inclusive information has the potential to support women during appraisal and help-seeking intervals and subsequently improve the timeliness of cancer diagnosis and treatment. 35The provision of culturally relevant resources on gynaecological cancer symptoms for Aboriginal and Torres Strait Islander people may help reduce the longstanding inequities in gynaecological cancer outcomes.

| Strengths and limitations
While this evaluation comprehensively included relevant government and non-government-authorised Australian websites, an international evaluation would have provided a more accurate representation of resources available to the general public.The evaluation of relevant  Cultural inclusivity of 3-5 items is considered moderately inclusive and 6-7 items as inclusive.e Webpage also linked to tools (e.g.symptom checker, service finder, question builder) which were not included in the formal health literacy assessment although these are likely to increase the actionability of the resource.f Small image included, in header/footer and typically alongside an Acknowledgement of Country.g Link to further information/support provided in the website menu rather than on the webpage itself.
T A B L E 4 (Continued) resources was conducted with validated tools (SMOG, FRE, PEMAT) with consistent use of one online calculator, as recent evidence suggests that automated readability scores vary across calculators. 36These validated tools have previously been used to assess the quality of information created for Australian women following the renewal of the Cervical Screening Program. 37Furthermore, the FRE has been used to assess the readability of online information for women from culturally and linguistically diverse backgrounds with low health literacy. 38While the cultural inclusivity tool used is not validated, no alternative evaluation tools are available for use within Australia.Future research is necessary to validate this tool and to identify and evaluate other resource formats such as videos and podcasts.

| What do the results mean?
As the identified resources were not deemed to be readable, understandable, actionable or culturally inclusive, it is important to highlight the impact this has on gynaecological cancer pathways.The Model of Pathways to Treatment illustrates the importance of patient cancer awareness and health literacy for timely help-seeking and diagnosis. 5Low health literacy is associated with poorer health and higher mortality, and complex health materials act as a barrier to health. 39Implications for research and practice include building the cancer health literacy of the Australian population in an equitable and inclusive way, as per the Australian Cancer Care Plan. 10Consistent with the National Women's Health Strategy priorities, this may include, but is not limited to, educating people where to go for trustworthy information and improving awareness of gynaecological cancer symptoms. 40There is some evidence that interventions to promote awareness of cancer symptoms delivered to individuals and via public education campaigns may be successful, 41 although further research is required to test the effectiveness of promoting help-seeking behaviours, 42 including for gynaecological cancer symptoms. 43mproving online health resources in accordance with readability, understandability and actionability recommendations for a diverse Australian population is required.For example, integrating visual aids improves understandability and benefits those with limited health literacy. 23A few identified resources included a question prompt list, which has been found to be effective during oncology treatment consultations. 44A question list may also be useful to ease the barrier to seeking help when women feel unprepared to know what to ask prediagnosis 7 and, as an actionable item, supports women to take steps that can benefit their own health.Ideally, resources should be (re)designed with input from end users; in doing so, resources may be more appropriate, acceptable and feasible to implement for the intended population.As there already exists many information resources about gynaecological cancer symptoms for Australian women, the first step of re-design through a co-design process is to identify high-quality resources (i.e.identified here) and then work with end users to identify elements of these that need to be changed, improved or maintained. 13This audit of resources will form the basis of such work.

| CONCLUSION
This study provides a summary of online gynaecological cancer symptom resources in Australia and highlights the inadequate quality of pre-diagnosis resources available.Online information should be designed with particular consideration to people with low health literacy and for a culturally diverse population.A simple increase in the quantity of information will not solve this barrier with the internet already oversaturated; rather, the modification of existing resources to align with Australian standards for quality information is urgently needed.Finally, we advocate for health information resources to be (re)designed in collaboration with end users to ensure their suitability, both in terms of health literacy and cultural relevance.

Australian Government and Non-Government Websites
Is the resource available in Aboriginal and Torres Strait Islander languages?6.Is the language used appropriate, relatable and relevant for Aboriginal and Torres Strait Islander peoples? 7. Does the resource include a contact (e.g.phone number/website) for further support for Aboriginal and Torres Strait Islander peoples?
3. Does the resource include Aboriginal and Torres Strait Islander designs/artwork? 4. Does the resource articulate evidence of leadership, involvement and/or governance by Aboriginal and Torres Strait Islander peoples/communities in its design/development? 5.
Summary of quality evaluation for included resources.Services (CRAICCHS), resource #106].The other resource was an easy-read booklet (Cancer Councils of Australia, resource #78).A further 19 resources met one or two items each for cultural inclusivity.No resource met six or more items to be considered culturally inclusive.Overall, no individual resource met all the benchmarks for readability, understandability, actionability and cultural inclusivity of Aboriginal and Torres Strait Islander peoples.Six resources met the benchmarks for readability, understandability and actionability and identified a range of gynaecological symptoms (covering between one and four of the five symptom groups).These resources were published by the Cancer Institute NSW (resources #25, #26, #27) and the Cancer Councils of Australia (resources #57, #62, #78).One of these six resources was deemed to be moderately culturally inclusive (resource #78).
T A B L E 3 b SMOG reading level of grade 8 and below is recommended.cFREscore of 60 or higher is considered easy to read by the general public.dPEMATscore of 70 or higher indicates the content is understandable and/ or actionable.eCulturalinclusivity of 3-5 items is considered moderately inclusive and 6-7 items inclusive; due to the small number of resources meeting any cultural item, a mean was not calculated.Health Evaluation of included resources.
T A B L E 4